Methods and systems for wireframes of a structure or element of interest and wireframes generated therefrom

ABSTRACT

The disclosure relates to systems and processes for generating verified wireframes corresponding to at least part of a structure or element of interest can be generated from 2D images, 3D representations (e.g., a point cloud), or a combination thereof. The wireframe can include one or more features that correspond to a structural aspect of the structure or element of interest. The verification can comprise projecting or overlaying the generated wireframe over selected 2D images and/or a point cloud that incorporates the one or more features. The wireframe can be adjusted by a user and/or a computer to align the 2D images and/or 3D representations thereto, thereby generating a verified wireframe including at least a portion of the structure or element of interest. The verified wireframes can be used to generate wireframe models, measurement information, reports, construction estimates or the like.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to, and the benefit of, U.S. patentapplication Ser. No. 15/683,703, filed Aug. 22, 2017, which claimspriority to U.S. Provisional Application No. 62/377,819, filed Aug. 22,2016. Both of these applications are hereby incorporated herein by thisreference in their entireties.

FIELD OF THE INVENTION

The inventions herein relate generally to improvements in wireframegeneration and/or verification. Wireframes provided from the inventivesystems and methods are verified representations of the structure orelement of interest. Such verified wireframes can be used to derivemeasurements, construction estimates, and other useful outputstherefrom.

BACKGROUND

Accurate measurements of structures, such as all or part of a buildingand associated areas, are relevant for the construction, remodeling,roofing, building envelope modeling, landscaping, insurance, VirtualReality/Augmented Reality, and gaming industries. For example, suchinformation is often used by professionals to create quotes, supportfield planning, make design decisions, measure inspected items, andspecify materials and the costs for both new building constructions, aswell as for remodeling or repair of existing structures. Landscapedesigners likewise require accurate measurements. In the insuranceindustry, accurate information about costs is critical to determiningthe appropriate premiums for insuring buildings/structures, as well asthe reimbursements due to a policyholder resulting from a claim.

As a specific, but non-limiting example, the construction and insuranceindustries require accurate measurements of various parameters relatedto roof structures, such as roof dimensions, pitches, penetrations,conditions, obstructions, surfaces, areas, vertex type (e.g., roofcorner, vent count, etc.), edge types, surface types (e.g., greyasphalt), and associated material costs. Traditionally, measurements ofstructures, such as a building roof, have been done by hand. In order toobtain accurate measurements of a building, a worker would be requiredto visit the building site to take and record roof measurements by hand.To ensure accuracy, the worker would be required to climb onto the roofto obtain measurements, thus leading to potential injuries. Complex roofconfigurations would result in less accuracy and/or more danger to theworker. In some cases, it is not even feasible to climb a ladder andmeasure on the roof due to high pitch (>10/12) or when a roof containsdamaged and unstable sections. Lastly, in many roofing situations, timecan be of the essence: a damaged roof must be replaced quickly to reduceor prevent damage to the interior of the building and its contents. Thismeans that the contractor that can return a roofing quote estimate thequickest might expect to be hired for the job, whereas those submittingbids later will lose the opportunity to gain the revenue.

Obtaining highly accurate measurements using manual techniques such astape measurement devices is very difficult to achieve on largestructures such as roofs, buildings, facades, roads, etc. When survey orfield crews go out to capture the measurements on a building, typically,the field sketch will be documented on paper. Validating the accuracy ofsuch measurements is even more challenging because there is noverifiable relationship of the field sketch back to the originalstructure, unless photos were taken at each endpoint of eachmeasurement. This means that if a different measuring crew or even thesame crew re-measures the structure, there is a very high probabilitythat the two sets of measurements will be different. Unambiguouslyestablishing relevant endpoints in a repeatable manner could enablecomparison to confirm accuracy, possibly with a high precisionmeasurement device, such as a laser scanner. Another way to indicate theendpoints is to physically mark each measurement endpoint on thestructure. However, there is no way in either case to confirm that themeasurements were actually taken from those endpoints on the physicalstructure. In short, such manual in-field measurements do not providehigh confidence in the results because typically the user cannot verifythe accuracy of the provided measurements. This means that any fieldmeasurement errors cannot be verified correctly (i.e., with highprecision) after they are taken, so it is very common for these errorsto translate through inflated quotes and material cost, resulting inwaste and reduced profit margin for the project.

Given at least the concerns of accuracy, time, and costs issues, mucheffort has been directed toward improving the generation of measurementsand the related cost and labor estimates of roofs. To this end, variousmethods have been proposed to generate roofing measurements withoutrequiring a worker to visit the job site to perform measurements but,rather, to provide measurements from remote locations. Several manualmethods have been proposed for this endeavor using aerial images of aroof from which measurements can be derived, as set out, for example, inthe background section of US Patent Application Publication No.20110187713, the disclosure of which is incorporated herein in itsentirety by this reference. Such methods are time consuming, requirehighly trained personnel and/or do not provide highly accurate results.To reduce the time and effort needed to generate wireframes and relatedoutput direct from aerial images, the '713 Publication proposes asemi-automated processing methodology. U.S. Pat. No. 9,501,700, thedisclosure of which is incorporated in its entirety by this reference,purports to provide a fully automated method of processing aerial imagesto generate roof models or roofing estimates. However, neither of theseautomated methods has obtained widespread use as of today because, quitesimply, accurate results cannot be assured. Again, the confidence in theoutput of such methods is not high.

In contrast to the direct output from aerial image methodologies set outin the '713 Publication and the '700 patent, point cloud information canbe used to generate roof measurements having improved characteristics.Such improvements are due, at least in part, to the point clouds beingmore likely to comprise useable 3D information, as compared to thedirect-from-image methods from which 3D information need to be extractedfrom two or more images after establishing correspondences among similarpoints in images. However, current methodologies do not readily allowpoint clouds that comprise structural information to be processedautomatically to generate accurate wireframes that can be used as outputsuitable for roof models, measurements, estimates etc. As a result, suchpoint cloud information must undergo substantial manual processingeffort to produce a suitable wireframe that can provide useful outputssuch as measurements, estimates, etc. This is generally accomplished byimporting a point cloud into a 3D CAD tool (for example, AutoCAD®,Revit®, etc.) followed by time consuming manual manipulation of thewireframe to generate a suitably accurate wireframe. Often, visuallyfinding 3D points to build a wireframe on a point cloud while zoomed inis challenging due to loss of perspective for the overall structure.Moreover, even with such extensive manual manipulation, accuracy ofoutput, for example measurements and/or geometric information, that arederivable from such point clouds cannot be guaranteed. In short,existing methods of processing point clouds to generate wireframes andrelated output do not provide high user confidence in the output.

Notably, any errors in the original wireframe output will be propagatedthroughout direct and indirect use of the output. For example, if ameasurement derived from the output deviates from the actualmeasurement, any construction estimate generated from that measurementwill also deviate from the actual requirement. Overestimation ofmaterials and labor due to uncertainties in measurement confidence is acommon process to ensure enough material is on site to complete the joband avoid delays in shipping of new material. Further, if the deviationis in regard to a roof pitch, the measurement mismatch, and anyassociated downsides, between the actual roof measurement and the roofmeasurement generated from an automated or manual output will beexacerbated.

Put another way, existing methods of generating wireframes andassociated output from structures or elements of interest, such asroofs, are time consuming and known to typically generate inaccurateresults, whether they be in the form of models, measurements, geometricinformation, construction estimates, reports, manual sketches etc. Suchknown inaccuracy results in a lack of user confidence in the results,and attendant compensation, such as by an additional validation orre-measurement step and/or increased labor and material costs, areneeded to counter negative outcomes related thereto.

In addition to being able to generate wireframes from which accuratemeasurements can be derived, current methods of generating wireframes donot provide users with any degree of confidence that the wireframe andany associated output therefrom are accurate representations of thestructure or element of interest. In short, users of wireframes arerequired to trust that the wireframe and any associated output are, infact, what they purport to be. Given the stakes involved in suchreliance, users will typically make educated guesses at how incorrectthe wireframes and associated outputs are, thus resulting in downsidessuch as ordering extra supplies “just in case.”

Moreover, in some cases, a wireframe and associated output that are onlymarginally accurate, may be “good enough” for some purposes. Usersnonetheless are not provided with any knowledge of how correct orincorrect the wireframe and associated output might be in a particularsituation. In other words, inaccurate wireframes are indistinguishablefrom accurate wireframes as outputs from the vantage point of the users.Verification that a wireframe is “good enough” can be relevant in someuser contexts, but there is no current methodology that provides theinformation needed to generate such user confidence in the nature andquality of a generated wireframe and associated output therefrom.

There remains a need for reducing and/or streamlining the amount ofeffort currently required to generate accurate wireframes of structuresor elements of interest. Still further, there is a need for improvementsin the ability to generate user confidence in the information derivablefrom such wireframes and output generated therefrom. The presentdisclosure provides these and other benefits.

SUMMARY

Aspects of the present disclosure are related to systems and processesfor generating verified wireframes. In one aspect, among others, amethod of generating a verified wireframe of a structure or element ofinterest comprises generating, by a computer or a user, an unverifiedwireframe corresponding to at least part of a structure or element ofinterest comprising one or more structural aspects of interest, whereinthe unverified wireframe: is derived from a plurality of overlapping 2Dimages of the structure or element of interest, wherein the 2D imagesare generated from a passive image capture device, and incorporates oneor more of the structural aspects; and comprises one or more featuresthat correspond to the one or more structural aspects; and projectingthe unverified wireframe over either or both of: one or more 2D imagesselected from the plurality of overlapping 2D images, wherein each ofthe selected 2D images incorporates at least some of the one or morestructural aspects; or a point cloud derived from the plurality ofoverlapping 2D images, wherein the point cloud incorporates at leastsome of the one or more structural aspects; and adjusting, by either orboth of the computer or user, at least part of the unverified wireframe,wherein the adjusting is by alignment of at least one of the one or moreunverified wireframe features with a corresponding structural aspectvisible in each of the one or more selected 2D images or the point cloudover which the unverified wireframe is projected, or indicating, byeither or both of the computer or user, acceptance of the unverifiedwireframe without adjustment, thereby generating a verified wireframecomprising all or part of the structure or element of interest.

In one or more aspects, the structure or element of interest cancomprise all or part of a roof, and the one or more structural aspectsof interest comprises a roof area, a roof corner, a roof pitch, a roofedge, a roof gutter, a roof gable, a dormer, or a skylight. At leastpart of the adjusting can be conducted by a user and a plurality of 2Dimages are selected for presentation to the user, wherein each of theplurality of 2D images are selected by the user or the computer topresent a different view of the one or more structural aspects ofinterest to the user. When the user aligns at least one unverifiedwireframe feature with a corresponding structural aspect of interest ona presented 2D image, modifications to the wireframe resulting from thealignment can be visible to the user on at least one other 2D image inthe plurality of presented 2D images. Measurement information can bederived from the verified wireframe, and wherein the measurementinformation can be less than 5% different from an actual measurementvalue for the corresponding structural aspect, or can be less than 1%different from an actual measurement value for the correspondingstructural aspect. In various aspects, the adjusting step can beassociated with one or more rules associated with the one or morestructural aspects of interest; and each of the one or more rules candefine an allowed positioning for the one or more structural aspects, aprohibited positioning for the one or more structural aspects, or bothan allowed and a prohibited positioning for the one or more structuralaspects. When a user generated adjustment conforms to an allowedpositioning for the structural aspect the adjustment can be maintainedby the computer; or when the user generated adjustment conforms to aprohibited positioning for the structural aspect the adjustment can berejected by the computer. The wireframe verification process can begenerated by a user, and during the projection and adjustment stepseither or both of visual or audible instructions can be provided to theuser by the computer. Information about one or more of the projection orthe adjustment of the unverified wireframe can be collected by thecomputer for incorporation into a machine learning process.

In another aspect, a system comprises at least one computing devicecomprising a processor; and a wireframe verification application storedin memory, where execution of the wireframe verification application bythe processor causes the at least one computing device to: generate anunverified wireframe corresponding to at least part of a structure orelement of interest comprising one or more structural aspects ofinterest, wherein the unverified wireframe: is derived from a pluralityof overlapping 2D images of the structure or element of interest,wherein the 2D images are generated from a passive image capture device,and incorporates one or more of the structural aspects; and comprisesone or more features that correspond to the one or more structuralaspects; and project the unverified wireframe over either or both of:one or more 2D images selected from the plurality of overlapping 2Dimages, wherein each of the selected 2D images incorporates at leastsome of the one or more structural aspects; or a point cloud derivedfrom the plurality of overlapping 2D images, wherein the point cloudincorporates at least some of the one or more structural aspects; andadjust at least part of the unverified wireframe, wherein the adjustingis by alignment of at least one of the one or more unverified wireframefeatures with a corresponding structural aspect visible in each of theone or more selected 2D images or the point cloud over which theunverified wireframe is projected, or indicate acceptance of theunverified wireframe without adjustment, thereby generating a verifiedwireframe comprising all or part of the structure or element ofinterest. The unverified wireframe can be generated based upon userinput.

In one or more aspects, at least part of the adjusting cam be conductedbased upon a user input and a plurality of 2D images are selected forpresentation to the user, wherein each of the plurality of 2D images areselected by the user or the computer to present a different view of theone or more structural aspects of interest to the user. When the useraligns at least one unverified wireframe feature with a correspondingstructural aspect of interest on a presented 2D image, modifications tothe wireframe resulting from the alignment can be visible to the user onat least one other 2D image in the plurality of presented 2D images.Measurement information can be derived from the verified wireframe, andwherein the measurement information can be less than 5% different froman actual measurement value for the corresponding structural aspect, orcan be less than 1% different from an actual measurement value for thecorresponding structural aspect. In various aspects, the adjusting stepcan be associated with one or more rules associated with the one or morestructural aspects of interest; and each of the one or more rules candefine an allowed positioning for the one or more structural aspects, aprohibited positioning for the one or more structural aspects, or bothan allowed and a prohibited positioning for the one or more structuralaspects. When a user generated adjustment conforms to an allowedpositioning for the structural aspect the adjustment can be maintainedby the wireframe verification application; or when the user generatedadjustment conforms to a prohibited positioning for the structuralaspect the adjustment can be rejected by the wireframe verificationapplication. The wireframe verification application can present eitheror both of visual or audible instructions to the user during theprojection and adjustment steps. Information about one or more of theprojection or the adjustment of the unverified wireframe can becollected by the wireframe verification application and stored in thememory for incorporation into a machine learning process. The indicationof acceptance can be based upon a user input.

Additional advantages of the invention will be set forth in part in thedescription that follows, and in part will be apparent from thedescription, or may be learned by practice of the invention. Theadvantages of the invention will be realized and attained by means ofthe elements and combination particularly pointed out in the appendedclaims. It is to be understood that both the foregoing generaldescription and the following detailed description are exemplary andexplanatory only and are not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow diagram illustrating an exemplary implementation of awireframe verification, in accordance with various aspects of thepresent disclosure.

FIG. 2 is a block diagram illustrating an example of a machine that canbe used for wireframe verification, in accordance with various aspectsof the present disclosure.

DETAILED DESCRIPTION

Many aspects of the disclosure can be better understood with referenceto the Figures presented herewith. The Figures are intended toillustrate the various features of the present disclosure. Moreover,like references in the drawings designate corresponding parts among theseveral views. While several implementations may be described inconnection with the included drawings, there is no intent to limit thedisclosure to the implementations disclosed herein. To the contrary, theintent is to cover all alternatives, modifications, and equivalents.

The term “substantially” is meant to permit deviations from thedescriptive term that do not negatively impact the intended purpose. Alldescriptive terms used herein are implicitly understood to be modifiedby the word “substantially,” even if the descriptive term is notexplicitly modified by the word “substantially.”

“Unverified wireframe” is a wireframe that could be derived from one ofthe following sources of input data: 1) a set of 2Daerial/terrestrial/satellite/orthographic/mobile/etc. imagery from thestructure or element of interest; 2) a 3D representation of thestructure or element of interest, where the 3D representation is derivedfrom point cloud information or surface mesh or computer aided design(CAD) model or building information modeling (BIM) model incorporatingthe structure or element of interest; or 3) combination of the twopreviously mentioned sources. The unverified wireframe can be generatedby a computer or a user, as set out in more detail herein. If theunverified wireframe is generated by a computer, such generation canoccur automatically. When generated by a user, the user can generate awireframe by extraction of the edges, vertices, etc., such as by drawingthese elements and others to create a viewable line drawingcorresponding to a viewable 2D or 3D image of the structure or elementof interest. The unverified wireframe comprises one or more features orelement that correspond to one or more structural aspects of a structureor elements of interest.

“Unverified wireframes” also include wireframes that comprisemodifications of the original unverifed wireframe, but for which theverification process has not yet been completed. Adjustment to theoriginal unverified wireframe can be made by comparison of one or morefeatures on the unverified wireframe with at least one 2D image of thestructure or element of interest or at least one point cloudincorporating a corresponding feature on the structure or element ofinterest or the combination of the two. The adjustment processes producea modified but as yet unverified wireframe that can be further adjustedduring the wireframe verification process. All or part of suchverification can be made by visual comparison by a user of theunverified wireframe with a portion or portions of one or more 2D imagesor point cloud of the structure or element of interest. Adjustments tothe unverified wireframe resulting from a determination that one or morefeatures of the unverified wireframe is substantially not in alignmentwith at least one corresponding feature in the image(s) or point cloudcan be made manually by a user or by the computer via machineinstructions as set out further herein, or a combination of both useraction and computer action.

A “verified wireframe” is the output of the wireframe verificationprocess. Such verified wireframe can be a modified version of theoriginal unverified wireframe, where such modification(s) incorporatesone or more adjustments made to the unverified wireframe in thewireframe verification process. The verified wireframe can also be anexact copy of the original unverified wireframe if it is determined inthe verification process by the user or the computer that the originalunverified wireframe needed no adjustment, either because the accuracywas desired and the unverified wireframe was deemed to be accurate asgenerated, or the generated wireframe was deemed to be “good enough” forthe intended use such that no adjustment was needed.

Adjustment to the unverified wireframe can be made by comparison of oneor more features on the unverified wireframe with at least one 2D imageof the structure or element of interest or at least one point cloudincorporating a corresponding feature on the structure or element ofinterest or the combination of the two. All or part of such verificationcan be made by visual comparison by a user of the unverified wireframewith a portion or portions of one or more 2D images or point cloud ofthe structure or element of interest. Adjustments to the unverifiedwireframe resulting from a determination that one or more features ofunverified wireframe is substantially not in alignment with at least onecorresponding feature can be made manually by a user or by the computervia machine instructions as set out further herein, or a combination ofboth user action and computer action. The verified wireframe can also bean exact copy of the unverified wireframe if it is determined in theverification process by the user or the computer that the unverifiedwireframe needed no adjustment. An unverified wireframe becomes a“verified wireframe” after going through the inventive verificationprocess herein even though no adjustment is made thereto.

An “accurate verified wireframe” is a specific type of verifiedwireframe that includes measurement and geometric information that issubstantially identical to that of the actual measurements and geometryof the structure or element of interest that is being modeled in thewireframe generation process. In other aspects, the verified wireframeis a substantially accurate model of the structural aspects of interestpresent in the structure or element of interest.

In some implementations, the verified wireframe need not be an accuratewireframe, as such accuracy is defined elsewhere herein. As one example,a “low quality” wireframe—that is, one that may not substantiallyconform to the dimensions of the structure or element of interest—can beverified by the user to be low quality. While such a low qualitywireframe may be suitable for some uses, user verification allows theuser to understand that the subject wireframe is likely not suitable forextracting accurate measurement information etc. therefrom. For example,the verification can allow the user to have confidence that addingadditional materials to his supply order is appropriate given the lackof accuracy known to be present in the measurement information etc.derived from the wireframe as a result of verification thereof.Verification of the wireframe as low quality can alternatively providethe user with confidence that the wireframe generation process must beredone, such as by acquiring new point clouds of the structure orelement of interest. Such new point clouds can be acquired by generatingnew images, point clouds etc. of the structure or element of interest,or by processing a different set of images, point clouds etc. togenerate a further wireframe for verification.

A “structure or element of interest” can encompass a wide variety ofstructures or elements that may be present in a scene such as, forexample, components of a building (e.g., doors, windows, trim, shutters,walls, roof, roof edges, gutters, stairs, plumbing/piping, electricalequipment, flooring material, decorative aspects), skylights, airconditioning units, landscape components (e.g., trees, shrubs,driveways, water features), vehicles, people, animals and the like.Indeed, a “structure or element of interest” can be anything from whichinformation suitable for processing to provide information about suchstructure or element(s) can be derived from a scene. Further, themethodology herein can be utilized to extract information about morethan one structure or element of interest in a scene, such as acollection of smaller structure or elements (e.g., doors, windows, etc.)associated with a larger structure or element (e.g., the overalldimensions of a building) where information about such collection ofsmaller and larger structure or elements can be processed and, in someaspects, the one or more structure or elements can be identified fromthe scene.

As used herein, the phrases “one or more structure or elements,”“structure or element of interest,” “structure or element(s) ofinterest” and “structure or elements” can be used interchangeably, andit is to be understood that, in some instances, one structure or elementof interest may be present or partially present in a scene, and inothers, there may be more than one structure or element of interest inthe scene. “Partially” (“or at least part of”) in reference to the oneor more structure or elements of interest refers to an instance whereonly a portion or portions but not the whole of a structure or elementis visible in the scene. In some aspects, the present methodology can beutilized to generate information about single or multiple structures orelements of interest that may be present or partially present in ascene, as desired by a user. An example of partial presence of astructure or element in a scene can include instances where thestructure or element is only partially captured due to occlusion or theperspective of the capture device or where parts of the structure orelement fit into the scene but the entirety of the structure or elementdoes not.

In some aspects, the present invention provides methods to verify andadjust, if necessary, an unverified wireframe that is derived from apoint cloud, where the point cloud is generated according to methodologyas set out hereinafter. In contrast to previous methods of verifying andmanually adjusting wireframes to accurately model the structure orelement of interest, the inventive methods and systems allow a user togreatly reduce the amount of time needed to complete the verificationand any associated adjustment tasks. In this regard, the inventorsherein have determined that visualization by projection or overlay ofthe unverified wireframe on or over a point cloud derived from thestructure or element of interest and/or on or over at least one 2D imagerepresentation or a 3D representation (e.g., derived from the pointcloud information) of the structure or element of interest results in averified wireframe. Such verified wireframe provides the benefit ofproviding user confidence in the accuracy (or lack thereof) of thewireframe that is so verified, and any output associated therefrom.

As set forth in more detail herein, such user confidence can result fromprojecting or overlaying the unverified wireframe on one or more 2Dimage representations and/or a 3D representation of the structure orelement of interest, and facilitating navigation by the user through andaround the unverified wireframe during a verification process. Suchnavigation can include zooming and panning to provide multiple anglesfor the unverified wireframe to allow the user to inspect the unverifiedwireframe vertices, whereby the positioning of such vertices can beadjusted against the 2D image representation(s) and/or 3Drepresentation. By verifying that at least the vertices of theunverified wireframe are or are not accurately matched or aligned withthe corresponding vertices on the 2D image and/or 3D representations, averified wireframe having user confidence in the output therefrom can begenerated. User confidence can also result from knowledge that a fullyor partially automated or manual wireframe verification and adjustmentprocess has been conducted according to a defined process configured toensure that the verified wireframe is or is not an accurate model of allor part of the structure or element of interest.

User confidence is the ability to trust or rely upon the informationprovided by the verified wireframe based upon desired criteria of theuser, without the need for further verification. For example, the usercan be confident that the wireframe endpoints are accurate to within theresolution provided by the one or more 2D image representations and/or3D representation, and thus measurements based upon the endpoints wouldbe similarly accurate. This resolution can be affected by, e.g., pixeldensity, distance, image angle, focus, etc. of the one or more 2D imagesprovided for verification. Yet further, the user can be confident thatthe wireframes and any associated outputs are not accurate in relationto one or more aspects of the structure or element of interest.Knowledge of whether the wireframe is or is not accurate can haveutility as discussed elsewhere herein.

The unverified wireframe that is verified and adjusted, if appropriate,in accordance with the present invention can be derived from a pointcloud, where the point clouds incorporate all or part of a structure orelement of interest. The point cloud will comprise one or morestructural features of interest present in the structure or element ofinterest.

As used herein, a “point cloud” is a set of data points in the samecoordinate system. In a three-dimensional coordinate system, thesepoints are usually defined by X, Y, and Z coordinates. There are avariety of methodologies to that can be used to generate wireframes frompoint clouds, and such are not part of the invention herein.

A plurality of 2D images can be captured and used to generate and/orverify a wireframe of the structure or element of interest. In someimplementations, these 2D images can be separated into different groupswhich can be independently used to generate the unverified wireframe andto validate that unverified wireframe after it has been generated. Inother aspects, some or all of the 2D images can be used to both generateand validate the unverified wireframe. Accordingly, at least part of theverification methodology comprises providing a first plurality of 2Dimages from which a unverified wireframe is derived via a point cloudgenerated therefrom and selecting a second plurality of 2D imagestherefrom and/or designating separate 2D images for use in the visualverification process. When one or more 2D images are used in theverification process of a wireframe that is generated from a set of 2Dimages, the 2D image frame on which the wireframe is projected oroverlaid comprises at least a portion of the structure or element ofinterest from which the unverified wireframe is generated. In someembodiments, only some of the 2D images that were used to generate theunverified wireframe and the associated point cloud may be used forverification and adjustment of the unverified wireframe. The selectionof the number and content of the 2D images can be made by the user orthe computer or both.

In some implementations, the 2D images used for verification need not bea part of the image set used for generating the unverified wireframe. Inthis regard, the 2D images used in the verification step could bederived from a set of images that were independently generated, that is,produced separately from, the 2D images from which the unverifiedwireframe was derived. Information that registers the set of 2D imagesused for verification to the unverified wireframe will be associatedtherewith, as discussed further hereinafter.

In some aspects, the verification processes herein can incorporate someor all of the following elements: a plurality of 2D images forgeneration of a point cloud; a point cloud (sometimes in a displayableformat) generated from such images; information regarding intrinsic andextrinsic camera parameters corresponding to each image, and/or otherinformation that describe the relationships between the images.

The inventive processes and wireframes generated therefrom can becontrasted to alternative approaches to manually creating a wireframefrom a point cloud using a CAD tool, in which a user manually selectspoints in the point cloud and identifies the subject points as verticesof a wireframe. In this regard, the inventive systems and verifiedwireframes are substantially not generated using a CAD tool and theverified wireframe and any associated downstream use thereof are notderived from a CAD tool. The inventive wireframe generation process isstreamlined and made faster because the user can select a vertex acrossone or more 2D images and/or a 3D representation of the structure orelement of interest rather than relying solely on the 3D point cloud fordirect generation of the wireframe. Still further, the inventiveprocesses and wireframes consist essentially of processes thatincorporate projecting or overlaying of the unverified wireframegenerated from a point cloud of the structure or element of interest onone or more 2D images or a point cloud of the structure or element ofinterest. When a wireframe intended for use in one or more downstreamprocesses is generated in this manner, a verified wireframe isgenerated.

Suitable point clouds from which the unverified wireframe can begenerated, or on which such first generated wireframe can be projectedor overlaid, can be generated from a plurality of 2D images generatedfrom a single passive image capture device. In this regard, point cloudssuitable for processing according to the methods of the presentdisclosure can be generated according to the methods disclosed in theU.S. Pat. No. 9,460,517, the disclosure of which is incorporated hereinin its entirety by this reference. In particular, a point cloud fromwhich the unverified wireframe is derived and that can be used togenerate a point cloud for the visual verification can be generated froma plurality of 2D images generated from a single passive image capturedevice. Yet further, the point cloud consists essentially of a pluralityof 2D images generated from a passive image capture device. Yet further,the 2D digital images from which the point cloud(s) are derived can begenerated by an image-capture device that consists essentially of apassive sensing technique. In some exemplary aspects, the inventivemethod herein can utilize a technique that processes a plurality ofstationary or video images to provide dense point cloud-based 3Dreconstructions of structures or elements comprising planar surfaces,such as roofs, followed by extraction of all vertices and/or edges ofthe included planes and creation of vertices on the generated wireframe.

The image-capture devices from which the 2D images are generated can beintegrated into a device such as a smartphone, tablet or wearable deviceor the image-capture devices can be as stand-alone camera device. Theimage-capture device can also be incorporated in a specializedmeasurement device.

An exemplary method that can be used to generate point clouds using thepassive image capture photogrammetry method of the '517 patent comprisesreceiving a plurality of 2D digital images of the scene, where all orpart of at least one structure or element of interest is present in ascene. The structure or element of interest can be selected from thescene by a user or by a computer. The 2D digital images will at leastpartially overlap with regard to the relevant structural features of theselected structure or element of interest. In order to generate a pointcloud for all or part of the structure or element of interest, at leastsome of the 2D digital overlapping images of the object are processedusing methodology that incorporates a structure from motion algorithm.

Point clouds derived from stereographic image capture methodologies canalso suitably be used herein. One type of stereographic imaging that canbe utilized to generate suitable point clouds for use herein aredisclosed in U.S. Pat. No. 8,897,539, the disclosure of which is herebyincorporated by reference in its entirety.

Point clouds derived from structured light imaging devices e.g., thefirst version of Microsoft Kinect®, Matterport®, Tango®, etc. can alsobe used. As would be understood, such devices combine RGB imaging withdepth detection otherwise known as RGBD images. Such images can beprocessed to generate point clouds using known methods, such asutilizing MATLAB, or open source software libraries, such as the “PointCloud Library.” Yet further, Tango-derived images incorporateinformation derived from motion tracking with integration ofaccelerometer and gyroscope data to generate detailed information aboutthe movement of the image capture device in space, as well as depthinformation about one or more objects of interest in a scene. Softwareconfigured for use with Tango-derived images can be used to generatepoint clouds therefrom. Other forms of structured light instruments andmethods can be used to suitably generate point clouds for use herein.

Point clouds generated from time of flight imaging devices are alsosuitable for use herein. As would be recognized, a time of flightimaging device computes the distance or depth value based on the knownspeed of light and based on measuring the time of flight of a lightsignal between the camera and the reflecting object, for each point ofthe resulting image. In a time of flight imaging device, the entirescene is captured with each laser or light pulse. The current version ofMicrosoft Kinect® is a time of flight imaging device.

Yet further, point clouds generated from ground-based or mobile orairborne or time-of-flight or phase-based LIDAR or other appropriatelaser scanner can be used herein. Laser scanning can utilize movablemirrors to redirect a laser beam to obtain measurement information oneor more dimensions. One suitable method for generating point clouds fromLIDAR is disclosed in US Patent Publication No. US 2009/0232388, thedisclosure of which is hereby incorporated by reference in its entirety.

Point clouds suitable for use herein can also be generated from GPSinformation coupled with provided 2D images. For example, when a numberof aerial images having suitable overlap are taken from multiple viewangles generated in conjunction with GPS information, a dense pointcloud of one or more objects present in a scene wherein the object(s)are geo-referenced can be generated. The aerial images can be obtainedusing manned or unmanned aerial vehicles such as, e.g., aircraft,drones, satellites, etc.

In the case of using 2D images along with a 3D representation of astructure or element of interest to obtain a verified wireframe, in someimplementations, it is not necessary for the image set to be involved inthe creation of the 3D representation of the structure or element ofinterest or even the unverified wireframe. For example, the unverifiedwireframe could be generated from a set of drone-captured imagery butverified using a set of aerial imagery or the combination of aerial anddrone-captured imagery. Another example would be to mix a set of aerialand ground-based (mobile) imagery for that purpose. Yet, another examplecould be using terrestrial laser scanning to generate a 3Drepresentation of the structure or element of interest but verify theunverified wireframe using a set of 2D imagery captured via a mobileimaging device such a DSLR camera. This could be even extended to usingan as-built BIM model to verify the unverified wireframe. The onlycriterion for this functionality is to register the 3D representation,unverified wireframe, and imagery in the same coordinate system. Thiscan be as simple as having the 3D representation, unverified wireframe,and different sets of imagery in a global coordinate system such asGeodetic or Geocentric coordinate system or different but known localcoordinate systems. A more complex scenario happens when the 3Drepresentation, unverified wireframe, and different sets of imagery arein different unknown coordinate systems. In this scenario, a set of 2Dand/or 3D features could be extracted and matched automatically amongthe different datasets or provided by a user. Corresponding 2D featurescan be converted into 3D features using visual triangulation techniques.Having at least three corresponding 3D features between two unknowncoordinate systems allows computing a transform matrix that maps the twocoordinate systems. Such a transform matrix or a set of transformmatrices could be used to bring the 3D representation, unverifiedwireframe, and different sets of 2D imagery into a known coordinatesystem.

Yet further, the present invention enables automaticadjustment/verification of the unverified wireframe based on new 2Dand/or 3D data that might become available in a later time. For example,a unverified wireframe for a building could be generated from a set ofaerial imagery that has been captured in the last winter. Because of arecent flight, a set of new aerial imagery with higher resolution orless-occluded viewpoints might now become available. The presentinvention could use this new dataset to automatically enhance the userconfidence and quality of the unverified wireframe to the highestfeasible accuracy level that can be achieved from the new imagery.

The present invention allows a user to obtain verified wireframes of allor part of a structure or element of interest by enabling quick andaccurate verification that the plurality of vertices and/or edges on theunverified wireframe match those same vertices and/or edges as appearingon a plurality of 2D images or are in alignment with 3D representationfrom which the unverified wireframe was derived. In other words,elements of the unverified wireframe that correspond to structuralaspects of interest on the structure or element of interest arealignable, and are aligned to generate a verified wireframe. The usercan also quickly adjust the unverified wireframe as necessary so as toallow verified wireframe to be generated. As would be recognized,verification of the correct placement of the vertices and/or edges onthe wireframe will translate into a verified wireframe that moreaccurately corresponds to the actual dimensions, geometry, andmeasurements of the structure or element of interest, for example aroof.

The verified wireframes can be used to provide useful information. Inthis regard, the outputs of the verification processes herein, that is,the measurements, dimensions, geometry, etc. for the structure orelement of interest have applications in a number of useful areas,including, but not limited to: inventorying, asset management,construction, merchandising, insurance underwriting and claimsadjustment, civil engineering, architecture and design, buildinginformation modeling (BIM), home remodeling, roofing, flooring, realestate listing, gaming, mixed reality, virtual reality, augmentedreality, among other things.

Moreover, the inventive process allows collection and analysis of usefulinformation generated during the verification and adjustment (if any)processes themselves or in user training, thereby enabling improvementsto be made in subsequent visual verification and adjustment tasks. Thefeedback given to a user by the computer during the user's verificationof the wireframe and any adjustments made thereto, can be useful totrain the user. In this regard, the wireframe verification processes ofthe present invention also can be useful improve the operation andassociated output of the user for subsequent wireframe verificationprocesses. In some aspects, the wireframe verification process willincorporate information about a level of experience of the subject user,where such information can further be utilized to develop userconfidence. For example, a user with significant experience in verifyingwireframes likely is more skilled at the task. Thus, it follows that amore confidence will exist in his wireframe verification than with auser who has less experience. The feedback associated with theverification and adjustment processes can be incorporated into orassociated with a user training module.

Any user interaction during the adjustment processes can be recorded andlater fed into a machine learning system to devise a computer model thatcan simulate/predict user behavior in verifying a wireframe. This couldinclude objective and/or subjective logics for removing/adding vertices,removing/adding edges, projecting vertices onto a specific surface suchas a planar surface, adjusting 2D/3D location of a vertex or edge,merging/splitting edges, merging/splitting surfaces, expanding/shrinkingedges, expanding/shrinking surfaces, intersecting edges or surfaces,estimating the geometry of the wireframe in occluded areas, labelingdifferent edge types, labeling different surface types, etc. Such usefulinformation can be used to train subsequent processes, to furtherstreamline and improve future verification processes. Such training canbe used for manual adjustment and/or those operated or assisted bymachine learning techniques, for example. Such verification processesresult in a verified wireframe having user confidence.

In some aspects, the visual verification and adjustment methods andsystems herein can be used to create a verified wireframe that issubstantially more accurate than the unverified wireframe, whereaccuracy is in relation to the actual dimensions (e.g., measurementsand/or geometry), when such actual dimensions are generated fromphysical measurement of the corresponding locations on the structure orelement of interest. In this regard, once the inventive verificationprocess is substantially completed, verified wireframes obtainedaccording to the inventive methodology will be substantially accuratebecause they allow measurements or geometry or other dimensionalinformation to be derived therefrom that have less than about 10% orabout 5% or less than about 2% or less than about 1% or less than about0.5% deviation those of the actual (e.g., “real life”) element orstructure of interest. For example, if a roof section on a house has alength of 50 feet, the verified wireframe section corresponding to thatroof length will be generated as ranging in length of from about 45 toabout 55 feet, or from about 47.5 to about 52.5 or from about 49 toabout 51 feet or from about 49.5 to about 50.5 feet. In someapplications, even greater accuracy than about 1% error can be obtained,such as within about 1 inch or within about 0.5 inches or within about0.25 inches of the actual measurement value of the element or structureof interest. Yet further, less than 1% or less than 0.1% error in theresulting measurement or geometric information may be obtainable.

The described inventive process can be auto-directed using a workflowprocess in that a user can be prompted to verify the accuracy of (or toadjust) the unverified wireframe in relation to the one or more 2Dimages or a point cloud from which the unverified wireframe was derived.The user can have the ability to observe the quality of alignment of theunverified wireframe with the one or more 2D images and/or the pointcloud in real time so as to allow confirm the correctness ofmodifications thereto while the visual verification and adjustmentprocesses are underway. During the adjustment process, the unverifiedwireframe is undergoing alignment with the one or more 2D images orpoint cloud and is undergoing adjustment.

While the amount of time necessary to visually verify and adjust (ifnecessary) a unverified wireframe using the inventive methodology willvary among users, generally, the method herein can allow verification ofan entire average roof wireframe (e.g., typical residential building) inless than about 10 minutes or in less than about 5 minutes or even lessthan about 2 minutes, where an entire roof wireframe comprises aplurality of individual wireframes that can be collected to provide theentire roof wireframes. When applied to verification of wireframesgenerated from roofs, the user will spend roughly less than about 5seconds or about 10 seconds or about 20 seconds or about 30 secondsverifying each vertex of the roof. Notably, current methodologies ofverifying and adjusting an average roof wireframe to make it accuratetakes users considerably longer time than with the inventive methods.Moreover, the subjectivity inherent in human-processed wireframemethodologies does not facilitate the generation of accurate wireframes,even when a user expends considerable time on verification. In short,the methodology herein enables not only faster verification ofwireframes derived from point clouds, but also results in more accuratewireframe generation (if that is a desired output), where accuracy is inrelation to the dimensions of the corresponding structural features ofthe element or structure that are represented by the wireframe. In otherwords, a partially or fully computer directed workflow process, such asprovided by the present invention, improves the accuracy of the outputwireframe. Moreover, when the user can observe the verification processand, indeed, is provided with the guidance to achieve such verificationas well as confirmation thereof, the user confidence is generated in theaccuracy of the verified wireframes. Alternatively, the verification canprovide user confidence that the verified wireframe and any associatedoutput is “good enough,” if that is the desired result.

For example, the present invention further allows an unverifiedwireframe that is derived from a poor quality point cloud or fromanother source to be user-modified to improve its accuracy, as well asto provide a verified wireframe and associated user confidencetherefrom. For example, when the unverified wireframe is generated by auser by drawing one or more parts of the structure or element ofinterest to be visible on the user's screen, that drawn unverifiedwireframe will likely be of a low quality, that is, will be lessaccurate, such as in relation to measurements, geometry or the like. Insome aspects, the methodology herein can enable a poor qualityunverified wireframe, such as the drawn unverified wireframe, to bequickly and easily corrected to convert it into a verified wireframethat is substantially accurate or otherwise suitable for the desireduse, that is, a wireframe that can be used output for use in CAD systemsor the like, so that output therefrom can be used to generate accuratemeasurements, reports, estimates etc.

The verification process can also be useful when the unverifiedwireframe is generated from a low texture scene, such as when the lackof features may not allow generation of a point cloud having suitabledensity and therefore fails to allow an accurate wireframe to begenerated therefrom. In this regard, during the verification and anyassociated adjustment of the unverified wireframe comprising astructural feature of a structure or element of interest, the user canbe directed or the user can direct himself, such as by user selection,to add additional vertices and/or to correct vertices and/or edges inrelation to the one or more 2D images of the structure or element ofinterest or in relation to the point cloud of the structure or elementof interest. In this way, the lack of coverage or overlap in sparse orempty regions of the point cloud can be compensated for in thewireframe. In some cases, the geometry of a more complete region of thepoint cloud (e.g., parameters of a plane) can be used to augment or fillin missing sections of the wireframe. Such verified wireframe (that is,the wireframe that has been verified, augmented and adjusted ifnecessary) will comprise dimensions that substantially match thedimensions of the structure or element of interest such that thewireframe can be used as an accurate model of all or part of thestructure or element of interest.

In a first element of application of the process of verification andaccuracy determination of a unverified wireframe, the user can bevisually presented on a display (e.g., a computer terminal) or suitableinteractive screen (e.g., a touch screen of a computing device) with a2D view of one of a plurality of images from which the wireframe wasderived or images that separately designated for verification, or wherethe coordinates of the images are registered with the wireframe. Yetfurther, a 3D view, such as by way of a point cloud from which thewireframe was derived can be overlaid with the unverified wireframe.Alternatively, a 3D view, such as by way of a point cloud, upon whichthe unverified wireframe can also be visually projected or overlaid canbe presented simultaneously with a corresponding 2D image view of one ormore 2D images. Still further, more than one of the plurality of 2Dimages can be presented to a user for overlay by the wireframe insequence, where the multiple images are presented as separate imageframes to the user.

While the description hereinafter specifically addresses verification ofa unverified wireframe where a roof is the structure or element ofinterest, it is to be recognized that the systems and methods herein canbe used to verify and adjust wireframes of other structures or elementsthat may be of interest. To this end, wireframes generated from a widevariety of structures or elements can be verified according to themethodology herein. In non-limiting examples, the present inventionallows verification of wireframes generated of roofs, skylights andother features on roofs, commercial/industrial building exteriors andinteriors, landscape elements, residential building exteriors andinteriors, floors, parking lots, roadways, bridges, machinery, or anyother suitable structure or element that may be of interest.

The processes and systems herein project the vertices and/or edges of aunverified wireframe onto one or more 2D images and/or a point cloudthat incorporate corresponding vertice(s) and/or edges to improve auser's ability to ensure that a wireframe generated therefrom is asuitable representation of the structure or element of interest, thatis, the wireframe generated therefrom is “verified.” The vertices and/oredges correspond to one or more structural features of interest in astructure or element of interest.

If the user determines that an adjustment of one or more features oraspects of the unverified wireframe is required to generate a desireddegree of alignment with the 2D image(s) or the point cloud, thatportion of the unverified wireframe needing adjustment is moved by theuser's dragging either or both of vertices and/or edges to align thosesame vertices and/or edges with the subject 2D image(s) or point cloudonto which the unverified wireframe is being projected or overlaid. Insome cases, dragging could mean clicking by the user at a new area orpoint or marked point on the image to choose the new vertex or edgelocation. Dragging an edge can also result in adjustment of theconnected vertex. When a vertex is dragged, the vertex can also bemarked on the screen or is otherwise associated with a user notificationto ensure that the user also verifies that the corner is correctlyconfigured after the line is moved.

To enhance the ability of a user to verify alignment of the subjectvertex and/or edge, auto-zooming functionality can be provided by thesoftware instructions to enable the user to quickly view image detailand make small adjustments to the wireframe. The process is configurableto allow the zoomed images to be moved or adjusted within the user'svisual field using zooming and panning functionalities.

Each of the one or more 2D images can be selected from the plurality ofimages that were used to generate the point cloud from which theunverified wireframe was derived. In other implementations, the 2Dimages can be from a group that was not used to generate the pointcloud, but was designated for independent verification of the unverifiedwireframe. For example, a subset of the full set of images that are usedto derive the unverified wireframe can be used in the visualverification process, a group of images that were not used to generatethe unverified wireframe can be used for verification, or a combinationof images that were used to generate the unverified wireframe and imagesthat were not used to generate the unverified wireframe can be used forthe verification. In this regard, the one or more images selected forvisual verification of a vertex can be those providing the largest angleof triangulation or can be those that are spatially closest to thevertex. Such larger angles of triangulation can enhance the ability of auser to visually verify alignment of vertices and/or edges between theunverified wireframe and those in the corresponding 2D image or images.Larger angles could ensure that the images that are most apart from eachother or images that show the same part of the structure but taken fromsignificantly different locations are used. The angle of triangulationcan be a significant feature because the larger the angle oftriangulation, the more confidence that can be generated from the 3Dcoordinates of the point (the region/volume of uncertainty is smaller).The angles can vary in this regard, but in some aspects, a suitableminimum angle of triangulation can be about 10 degrees or greater.

In further aspects, vertex and edge projections for the wireframe duringadjustment are kept substantially synchronized among one or more of theplurality of other presented 2D images as the user makes adjustments tothe unverified wireframe in relation to a first presented 2D image. Inother words, an adjustment made based on one of the 2D images is alsoapplied to the other 2D images used for verification. This can allow theuser to better visualize the impact of an adjustment to a single part ofthe wireframe as that adjustment propagates through the plurality of 2Dimages that incorporate different views of the structure or element forwhich a verified wireframe is desired. For example, if the user adjustsa vertex to align with a single 2D image, such adjustment may create alack of alignment in another image. Consider a unverified wireframe of aroof having vertices labelled A, B, C, D and E. Those vertices willappear in different perspectives in the underlying 2D images, and theimages can be identified as relating to one or more of the labelledvertices. The subset of 2D images that comprise suitable angles oftriangulation, as set out hereinabove, can be automatically selectedfrom the larger group of 2D images for presentation to the user forverification and, if appropriate, adjustment. This selected subsetbeneficially comprises one or more 2D images of the structure or elementof interest where the pictured vertex/vertices of the structure orelement of interest comprise angles of triangulation for thevertex/vertices that are suitably large enough to allow fordetermination of whether a corresponding vertex and/or edge of theunverified wireframe is accurate with regard to the dimensions of theactual roof. An image that contains more than one vertex—for example,vertices A and B—may present one vertex—here A—that has a suitable angleof triangulation, while the other may have too small of an angle oftriangulation with regard to vertex B. Only the image relevant to vertexA will be presented to the user with regard to verification andadjustment of the corresponding portion of the wireframe. In thissituation, that 2D image will be identified as relevant to verificationof vertex A, but not for vertex B. If a 2D image includes an angle oftriangulation that is suitable for more than one vertex, for exampleboth vertex A and vertex B, those images will be presented to the useras each of the respective vertices are being visually verified andadjusted by the user. In a situation in which multiple new verticesshould be added to a unverified wireframe in the same area of the objectof interest, the inventive systems and methods can provide the abilityto add—that is, triangulate—multiple new vertices on a single set of 2Dimages in a single or in multiple processing steps to streamline theoverall process. As long as the angle of triangulation of the multipleimages is sufficient to resolve each new point, the same images can beused for these new points.

Unlike prior art methods that generally rely on user-directed selectionof the point cloud for creation of wireframe vertices and for selectionof those wireframe elements once created for verification and selection,in some implementations, the present invention can provide users with asystematic and logical workflow process within which to visually verifyand adjust the various vertices and/or edges of a wireframe amongst andbetween various 2D images and/or a point cloud presented to the userduring the verification process. The present invention can provide theability to visualize the unverified wireframe and its constituentcomponents (e.g., vertices and edges) both in the unverified wireframeand as the unverified wireframe is projected onto the one or more 2Dimages and/or a point cloud.

The visual perspective of seeing a component in 3D form in theunverified wireframe, in addition to the ability to see how it exists in2D against one or more images of the actual object of interest or in 3Dagainst the point cloud from which the unverified wireframe was derived,can enable the system to walk (or direct) the user around the structureor element of interest in a logical flow while presenting images foreach vertex or area of the structure or element (e.g., bringing together3D to 2D views to provide the capability for fast and accurateprocessing). The addition of 3D processing during verification alsoenables the user to identify holes or missing areas of the wireframethat need to be addressed in order for the wireframe generated from thevisual verification and adjustment process to be accurate. In situationswhere addition of new vertices is indicated, or the entire wireframeneeds to be constructed, the 3D presentation provides the ability tochoose points in an intuitive manner, and the 2D images can provide theability to accurately select or create the point on the actual structureor element of interest. Such user involvement in the verification of theunverified wireframe generates user confidence in the resulting outputand any uses thereof.

The inventive process can substantially automatically provide, forlogical presentation to a user, one or more 2D images and/or a pointcloud that can facilitate verification so as to generate an accuratewireframe of all or part of a structure or element of interest. That is,2D images or portions of a point cloud that are relevant to vertex Awill be collected and presented to the user together. This canfacilitate efficient visual verification of vertex A and any adjacentedges because all, or substantially all, of the 2D images relevant tovertex A and/or those portions of the point cloud that are relevant tovertex A can be presented to the user at the time that the wireframeelements corresponding to vertex A are being visually verified andadjusted.

Notably, how an adjustment to one aspect of a wireframe will affectanother element of the wireframe is relevant in that a first adjustmentmay cause another wireframe element to become misaligned. By presentingthe user with a logical selection of all or part of one or more 2Dimages or the point cloud sequenced as a workflow, user subjectivity canbe reduced or even substantially eliminated, thus compressing the timeneeded to verify and adjust, if necessary, a unverified wireframe togenerate a verified wireframe. Such verified wireframe and provides userconfidence in the verified wireframe and any output and use thereof.

The unverified wireframe will be modified as the verification and anyadjustment process proceeds. The original unverified wireframe (that is,the generated wireframe) and any subsequent modifications thereto can bemaintained in memory. If mistakes are made during adjustment by theuser, the changes can be discarded and the original unverified wireframecan be recovered. A wireframe undergoing adjustment will generate aplurality of wireframe variations prior to generation of the verifiedwireframe at the end of the verification and adjustment process. Each ofthe plurality of wireframe variations can, independently, be saved inmemory for later recall or evaluation of each of the individualvariations.

The invention herein also can be configurable to maintain a log of dataregarding the distance of the user's manual adjustments for each vertexwhich can provide information related to the accuracy of the data thatare used to generate the unverified wireframe. Such accuracy informationcan be utilized to generate improved algorithms and associated methodsfor use in generating wireframes automatically or semi-automatically foruse at a later time.

In some aspects, the various verification and adjusting steps can beperformed according to a timed sequence. For example, the process can beconfigurable to require the user to act on verification and adjustmentof a unverified wireframe in relation to one or more 2D images and/or apoint cloud in a predetermined time, for example, less than about 5seconds or about 10 seconds or about 20 seconds or about 30 seconds pervertex, to be followed by the auto-advancing of the visual user cue(cursor or otherwise) to the next required verification element, such asthe next roof vertex where the unverified wireframe undergoingmodification is placed over a next relevant 2D image or over anotherpart of the point cloud corresponding to that next roof vertex. Suchauto-direction can facilitate quick and decisive action by users toenable them to meet production quotas. In some aspects, the user canoverride the auto-advance to enable the user to ensure that averification and possible adjustment step is conducted accurately.Whether or not auto-advancing is enabled, the system tracks and logs andoptionally reports time spent on each vertex, as well as any actionsperformed (e.g., adjustments, zooming/panning, etc.) Yet further, asequence can be put on verification and adjustment that is based on alocation and/or direction of points. For example, a user can start withthe most northern vertex in the wireframe and adjust one by one tillreaching the most southern vertex.

The verification and adjustment step can also be untimed, but once thestep is completed, the process can be configurable to auto-advance tothe next relevant 2D image and/or another location of a point cloudsubstantially without user input. The removal of the requirement of userinput can improve the user's attention to the verification andadjustment of the unverified wireframe. In this regard, the user can beable to maintain focus on the unverified wireframe undergoingmodification to generate a verified wireframe, which can further reducethe time required to obtain the associated verified wireframe in whichthe user has confidence.

In further aspects, the verification and adjustment process can beconfigured to execute in varying modes, however, the user may not beprovided visibility to the underlying mode being executed at that time.That is, each of the verification and adjustment steps can be configuredto align with an optimized workflow. The various operational modes cancomprise one or more of the following (which are discussed in moredetail in the workflow section hereinafter):

-   -   Inspection/Verification mode—the system is configured to allow        user review of each vertex and edge. User can review fit of a        unverified wireframe to a plurality of relevant 2D images and/or        a point cloud, and user can select any vertex or edge on the        wireframe undergoing verification and adjustment to see images        related to that selected vertex or edge. This mode allows the        user to verify a unverified wireframe through projected location        of a desired vertex in different images and projected direction        of all edges connected to that vertex in the same images. The        combination of projected vertex location and projected edge        directions generates sufficient mathematical constraints for a        very high confidence output.    -   Adjustment mode—the system is configured to allow user review        and adjustment of each vertex and edge.    -   Add element mode—the system is configured to allow user        selecting a point in an wireframe to generate a new vertex or 3D        point in the unverified wireframe; a related Connect mode allows        a user to connect multiple vertices to created wireframe edge.    -   Create mode—the system is configured to generate a new wireframe        or to add a new section to the wireframe by user selection of        points and adjustment of corners. Note that the names of the        modes are exemplary and not meant to be fixed in name.

In some cases, areas of the structure or element of interest may befully or partially occluded by other objects. In a non-limiting roofingexample, this could happen when trees obstruct a portion of the roof andthe generated point cloud including the structure or element of interestwould not include that portion of the roof (but may include the tree)and thus the resultant unverified wireframe will not include thatportion of the roof. In another example, a sign could obstruct a portionof the front of a building and the generated point cloud and theunverified wireframe will show the sign but not the building area behindthe sign. In these and similar cases, the inventive methodology canautomatically detect and highlight the occluded regions based on thepotential discontinuities in the geometry of the structure or element ofinterest. Once these areas are detected, pre-defined or templatedactions could automatically be implemented to compensate for theoccluded areas. In case of a roof wireframe as a non-limiting example,this could translate into creating a new edge that is parallel to theopposing edge in the given roof facet and extending the new edge so thatit intersects with neighboring edges. The inventive methodology can alsoenable the user to auto-extend and intersect the unverified wireframeedges to create a vertex. The inventive methods could then better ensurecertain geometries are met such as the 3D point exists on the relevantplane(s). In other situations the user could choose points near theedge(s) of the obstruction(s) and specify to extend beyond that point tointersect. The system can then optionally indicate the properties ofthose extended edges and vertices as lower confidence or obstructed orsome other defining term by coloring them a certain way or noting themwith a mark or describing them as such either in the verified wireframeor 2D images or point cloud or output report or generated outputformats.

The system is configurable, and options area laid out in logicaleasy-to-follow process, which may include one or more of the followingcharacteristics:

-   -   Wireframe overlay on|off—if wireframe overlay is off, then tool        acts in inspection mode, pick a point or edge and see zoomed        images for that point or edge, if wireframe on, will see        wireframe projected onto images    -   Number of projection views to display—system can default to        three projection displays—that is, relevant 2D images and/or        point cloud over which a wireframe is projected in a single        screen pass. If the user wants to verify wireframe        characteristics during each verification step and/or after        adjustment from more angles, the number of projections can be        increased, such as to four. User can choose to show fewer        projection views, one potential reason could be if unverified        wireframes are very accurate and want to speed the process.    -   Inspection mode|Verification mode—inspection mode allows user to        select any vertex or edge and will conditionally display zoomed        images of that vertex or edge.    -   Automatic or Selection mode—when in verification mode and in        automatic selection mode, the system starts at a vertex or edge        and ‘walks’ user around entire structure or element of interest        (e.g., the entire roof). This will present the user with each        vertex or edge, followed by moving to next vertex or edge after        adjustment, if necessary.    -   Automatic selection mode can also be timed for auto-advancing        based on time.    -   Selection mode allows user to select vertices or edges for        verification and adjustment in any order.    -   Input model(s): Filtered        Dense|Dense|Edge|Sparse|Voxel|etc.—allows different point cloud        types to be evaluated depending on data type and complexity.        Primary flow is a filtered dense cloud, but for very large        clouds, might want to enable edge cloud mode, and potentially        sparse or voxel for further analysis or for debugging.    -   Manual|Automatic Projection Views—when in verification mode,        automatic is default mode. Manual allows users to select which        images/camera poses to use as projection views; this mode can        optionally be turned on for one vertex or edges only in cases        where auto-selection of projection views is not ideal and want        to replace one or more of the views.    -   Zoom levels—system can be configurable to zoom to the highest        zoom level without loss in sharpness; range can allow for 0-100%        zoom.    -   Modules/Rules on/off—specific modules with specific        functionality or rules enforcement can be enabled or disabled.        For example, one rule could limit the adjustment or movement of        a vertex or edge to a predefined surface (e.g., a planar surface        that represents a roof facet).

The inventive tool can incorporate selectable modules that may comprise,among other things, properties, products, configuration, features, andconstraints that apply to specific applications, group of functionality,type of industry, type of customer, or type of structure or other. Forexample, a module could be specific to flooring vs. roofing, commercialvs residential roofing, sloped vs low-sloped/flat roofing. The modulescan contain many different types of information. In some embodiments,the information could be grouped or organized into properties,constraints, products, configuration or other types. The information inthe module impacts many aspects of the system.

Examples of functionalities that could change, and thus be selectable,according to a given module are: application-specific standardlabeling/terminology, algorithms to calculate pertinent geometricalproperties (surface area, perimeter, angle), methods/rules to extractsemantic properties (edge types, material type, manufacturer, heattransfer index, etc.), identify/label structure types, imposeconstraints, methods to extract metadata and shape data for specificproducts (e.g., 16″ wide×20′ long, standing seam metal panel, blue incolor, wind uplift 100 mph, CRS 16 gauge), etc. Other examples includeprogrammatic settings that the tool uses to perform automated featureextraction (e.g., a certain color threshold which is used for a featurelike determining texture type). A module could also dictate a certainuser interface by loading certain features (action buttons). Moreover,different types of constraints could be loaded to the tool based on thecurrent active module including: 1) a geometric constraint that enforcesor limits connectivity of an object to other objects; 2) a semanticconstraint that enforces or limits semantic relationships amongconnected objects (e.g., an edge with a certain edge type cannot beconnected to another edge with an edge type that does not conform to thefirst edge type); 3) constraints that describe relational connectivity(e.g., a floor object is enclosed by walls, a roof structure must beabove ground plane, etc.); 4) constraints to enforce the structuralintegrity of a building or building element; 5) constraints for clashdetection; and 6) constraints for detecting inconsistency amongdifferent levels of information.

In some aspects, verification and adjustment modules with differingfunctionality or rules enforcement can be enabled or disabled, in someimplementations. For example, a Roofing-Construction module can compriserules that confirm and enforce typical pitches and/or inner angles of apolygon representing a roof facet. Such a module could includefunctionality or parameters to turn on functionality like generating aDXF file from a button or automatically from the inventive tool. Thesystem could also include rules specific to an industry or structuretype.

Such rules can define allowed positionings during the unverifiedwireframe adjustment process. Such rules can further define prohibitedpositionings during the unverfied wireframe adjustment process. Suchrules can also define warnings or caution flags to the user during theunverified wireframe adjustment process. Prohibited positionings will beprevented from occurring as to attempted user adjustments thatcorrespond to the prohibited positionings. If the intended useradjustment corresponds to a defined warning or caution flag, the usercan be prompted to check her work to ensure the intended adjustmentmakes sense in context.

A rule-based model can be implemented once the adjustment is completedso that the verified wireframe can be verified to be geometrically validaccording to construction rules and conventions (e.g., symmetry, innerand outer angles, length, slope, number of edges in a fact, connectivityamong different facets, etc.). Still further, the verification andadjustment process, as well as the wireframe creation process, can beassociated with a look-up table where geometric shapes (or “geometricprimitives”) are stored. The characteristics of a verified and adjustedwireframe—that is, a second or accurate wireframe—can be compared to theinformation in the look-up table to provide a further verification ofthe accuracy of the generated wireframe.

Such a rule-based model that incorporates geometric shape informationcan also be used during the verification and adjustment process toprovide guidance to the user as the wireframe is being modified duringthe visual verification and adjustment process progresses. This guidancecan facilitate decision making by the user, thus further reducing usersubjectivity and potentially speeding up the verification and adjustmentprocess from which the accurate wireframe is generated.

The system can be configurable to display a verification level or statein a status area or other named pane while a verification and adjustmentactivity is being conducted with an unverified wireframe. Innon-limiting examples, information included in this status pane caninclude, for example, information about matters such as: status ofwireframe loading or being saved, total vertex count; number verified;number remaining; percentage of wireframe complete; configurationoptions (e.g. zoom level, mode, etc.). Summary reports can also begenerated for each verified wireframe, and its associated structure orelement (e.g., roof, door, building, etc.). In non-limiting examples,these summary reports can include data such as all data generated duringthe session, including number of vertices; number adjusted; amountadjusted (e.g. average/minimum and maximum per unit time); edgesadjusted (e.g. average/minimum and maximum per unit time); number of newvertices created; time on vertex (e.g. average/minimum and maximum perunit time); areas per plane/total area; time in model; listing of eachcorner with statistics for each vertex, etc. The data associated withthese and other status characteristics can be used to create activityreports for each user, such as to evaluate her productivity, error rate,etc.

A variety of data structures can be incorporated during processing.These can include the following as well as others:

-   -   Corner image identification, sequence number, initial location,        new location, time in focus, number of available views, status        (e.g., needs to be viewed, has already been verified, number of        times viewed, number of times adjusted), etc.    -   User action log—for example, every step a user makes or action a        user takes can be recorded to assist in streamlining processes        in generating process improvements    -   Session—point cloud name/size, total vertices, number of        vertices verified, number of vertices created, time in session,        configurations parameters for session, etc.

Various features can optionally be incorporated into the methodologyherein. For example, the process and system can be configurable to takeand record snapshots (e.g., part of the relevant 2D images and/or pointclouds where vertices or edges or surfaces were incorporated) of allverified vertices or edges or surfaces of the unverified wireframe andany adjustments thereto. Each verified wireframe can also be recorded assnapshots. All or some of such recorded snapshots can be used in otherprocesses, such as the training of machine learning algorithms that canbe used in future applications where additional automation may beindicated using such trained algorithms. Such information can also beused to support training of users or audits of completed verifiedwireframe generation processes.

Further features can optionally incorporate a mode that limits a user tocreate a vertex or edge only in places that a corresponding 3D dataexists in the 3D representation of the structure or element of interest.While adjusting in such a scenario, the user could drag a vertex or edgeto a pixel that is not associated with an accompanying 3D data and thesystem can be configured to triangulate based on those image coordinatesand create a new vertex or edge. This new vertex or edge can begenerated by the system according to geometrical rules associated withdistance to a 3D plane that is acceptable. Optionally, the newvertex/edge or to be created vertex/edge could be evaluated inassociation with shapes located in a look-up table as discussedelsewhere herein. The triangulation process could be performed inreal-time and hence the user could see the impact of adjusting thelocation of a vertex/edge in an image by immediately noticing themovement of the vertex/edge in 3D or movement of the projectedvertex/edge in other 2D images. If triangulation is found to be aresource-intensive process, the system can be configurable to restrictthe triangulation frequency.

Yet further, individual surfaces/polygons in an unverified wireframe canbe highlighted so that the adjustments will only be applicable to thesurface/polygon of interest. In this regard, the system can beconfigurable so that the adjustments will not propagate beyond affectingthat surface/polygon. In some implementations, this can be accomplishedat the vertex level by not connecting (or deleting) vertices to thedifferent polygons.

Vertex or other feature highlighting in the wireframes can also beconfigurable in the systems and methods herein. In this regard, thehighlights can be identified, such as by circling or marking eachwireframe so it is apparent to the user where she should select when inthe selection mode. A check mark or other signal can be placed in thecircle or mark or in place of the circle of mark when the user completesthe verification of that aspect of the wireframe. For corners that areadded by a user, the software can be configured to indicate they are notyet connected. Other forms of highlighting can also be used to assistthe user in verification and adjustment.

Additional features can be incorporated in the process to deal withdifferent types of penetrations or conditions that might exist in astructure. These features could include automatic detection ofpenetrations or conditions in roofing, for example, or regions ofinterest both in the 3D representation of the structure and the 2Dimagery taken from that structure. Computer vision, image processing,and/or machine learning algorithms could be used to detect and identifysuch regions if the algorithms were trained for those specific regions.In case of a new region type that prevents automatic detection andidentification algorithms to perform accurately, the user might manuallyselect a region of interest. The algorithms then automatically extractdistinctive 2D and/or 3D features from that area and look for the samefeatures in other parts of the 3D representation data and/or 2D imageryso that they detect and identify any other areas with similar features.As a non-limiting example, the user can select an air conditioning unit(a condition) in an image from a commercial roof structure. Onceselected, the algorithms will extract features from that region and as aresult will be able to detect and identify all other similar airconditioning units on that roof both in 2D and 3D datasets.

Machine learning algorithms can be trained using information generatedin the verification process, as well as any downstream applications thatutilize the verified wireframe and any output generated therefrom. Forexample, a training set can be used in conjunction with machine learningalgorithms in assigning properties and property weights, where “weights”refers to the confidence or significance assigned to a given propertyassociated with one or more aspects of the unverified wireframe andgeneration thereof, one or more aspects of user's actions during theverification process, aspects associated with the information andfunctions presented to the user during verification, or use of theoutput.

Each machine learning model will be trained following the traditionalsupervised learning approach or a more recent unsupervised learningapproach. As previously established, a training set of the data can beincorporated into instructions associated with subsequent verificationoperations. Training then involves allowing the algorithm to take in theinput information and learn the necessary model parameters that willassociate the input features or appearances to their corresponding labelvia the algorithm. Examples of such algorithms include, but are notlimited to the use of SVMs (support vector machines), randomizeddecision forests, convolutional neural networks and adaboost. Each ofthe aforementioned algorithms can be optimized according to theirrespective loss functions. A non-limiting example of a general lossfunction is argmin_(δ) max_(θ∈Θ)R(θ,δ). It is commonly referred to asthe min-max loss function. It determines the best δ minimizes themaximal possible loss θ. Other qualifying methods can be employed toevaluate the performance of the system. These include but are notlimited to the system's accuracy, recall, and error. Such methods can beemployed to improve the robustness in the algorithm's predictivecapabilities by means of cross-validation.

Multiple machine learning models can be trained and used. This isreferred to as a “mixtures of experts” or “ensemble,” where each modelis trained on all or part of the training data. Furthermore, anadditional layer of logic can be applied to the outputs of each methodto determine a weighting function that would give priority based on theperformance of each method.

In some aspects, therefore, the new additions to the training sets canbe made from the verification processes for the unverified wireframesthat are predicted via one or more machine learning algorithms and canbe validated with light supervision in which a human may readily“eye-ball” the results to confirm correct results or throw out badresults. As the machine learning algorithms are further trained as totype and characteristics of the structure or element of interest, thewireframe verification process can become substantially unsupervised, asper the semi-supervised approach. This can greatly accelerate thegeneration of verified wireframes of structures or elements of interest.Even with substantially unsupervised processes, from time to time, ahuman, optionally, can be used to validate at least some of theautomatically object predictions. Over time, the machine learningalgorithms can learn characteristics of one or all of the wireframegeneration rules, user operation, workflow and the like with lightsupervision or even substantially without the need for humansupervision, and the resulting information can then be used to performefficient and accurate object recognition-related activities for newlyprovided object recognition information, even those for which little orno substantive information about the object or structure of interest isknown beforehand, as long as the information relevant to the wireframegeneration and subsequent processing steps are collected, recorded andincluded the training set used with the machine learning algorithms.Such machine learning algorithms can be updated from time to time orcontinuously to result in further improvements in the processes andoutput related thereto. As would be recognized, the machine learningpredictions can be improved through continuous updates in the trainingin both a supervised and semi-supervised manner along with additionaltraining for the models on the additional information. Thus, it isanticipated that, over time, inclusion of new information generated fromthe methodology herein will improve subsequent verification operations.

The fully or partially adjusted unverified wireframes generatedaccording to the methodology herein can be utilized to generate CADinformation files that can be utilized to create, for example, 3D modelsof a location where such models may be used to generate architectural,construction, engineering, and other documentation related to aconstruction project. Both interior and exterior survey and measurementinformation can be provided for use for these purposes, as well as forcreating content for a virtual reality model or digital asset or othercomponent of a virtual reality experience.

The inventive systems can provide a wide variety of presentation stylesand formats of resultant verified wireframe or information associatedtherewith. Consumers of this information can be in the form of a user ora computer system. Users typically focus on visualization, layout, thetype of data, sometimes coupled with a need to modify the data to fitthe user's expectation. Systems sometimes require a focus on themechanism (in nonlimiting examples, API, cut and paste, monitoredeFolder), the language of the data (e.g., CSV, XML, geoJSON, JSON, freeform), the schema (structure of the metadata/element type) and the datatype format of the data itself (e.g., string, enums, etc.). In onembodiment the output can adhere to industry accepted or standardformats such as, e.g., DWG, DXF, STL, or others. The system also canprovide as output the visualization of the verified wireframe in a 3Dview. The transmit method can occur in different ways to supportdifferent application needs by the user. The full 3D view could betransmitted by uniform resource locator (URL) via an email or text. TheURL approach would allow the user to open and view the 3D View. Largefile sizes like a 3D point cloud may involve web chunking of the data toprovide visualization of the point cloud prior to the entire data setbeing downloaded. The system can also provide a read only 3D view thatlimits users to rotate, pan, zoom to visualize different aspects of the3D model. This “read only” approach would not support changes to thestructural data. Much like a printed output or output in the form of animmutable PDF file, the integrity of the results cannot be changed bythe user.

The system can also provide an animation capability to inject ananimation sequence into the output to better control the experience forthe user, possibly ensuring the optimal visibility from a prescribed setof angles. The system can provide real time data access by enablingembedding of the 3D view into a users' application or web site thataccepts refreshable data feeds. The system can provide the output inprintable form, for example as PDF or image or import into a wordprocessor or other presentation tool. The system can support bindabledata structures to enable potential combination data of output data withuser provided data. This would support customizable representation ofthe output, possibly adding user branding, generating estimates, orother forms of reporting. Furthermore, the output could be segmented orclassified to output a specific set of conditions or types ofstructures.

When a roofing-related information is the desired output, the outputcould consist of a roofing report, a gutter report, a siding report, avolume/square footage report, paint report, cost estimation, or otherreport. The system can also output the data in a raw format, thusenabling downstream calculations to create new data. In this approachthe system could output the data in a 2D tabular format (like CSV, EXL)or formatted data (JSON). When processed according to the inventivemethodology herein, verified wireframes of the structures or elementsare substantially accurate as explained elsewhere herein.

Workflows for the inventive systems and processes can be exemplified bythe below-described steps that are presented in relation to generating averified wireframe for a roof as a non-limiting example:

An example of the verification process is illustrated below. The belowprocess can be conducted using a computer, tablet, smartphone, or otherappropriate computing device.

Vertex Verification and Adjustment Workflow, a Roofing Example:

User clicks on link, browser or display view, which can open with:

-   -   Overlay model of, e.g., a house with a unverified wireframe        (possibly just roof planes with the unverified wireframe, or        possibly an orthogonal with the unverified wireframe). The        overlay model is the point cloud in a 3D viewer with or without        the unverified wireframe displayed.    -   Projection views of a first roof corner to evaluate that        corner/vertex (e.g., 3 or 4 frames zoomed to the corner, with a        corresponding unverified wireframe vertex and edges projected        onto a 2D image or photo). Projection views are the 2D images        with the wireframe projected correctly into the 2D image and/or        a point cloud from which the unverified wireframe was derived.        Or cases when creating the unverified wireframe, the 2D        images/point cloud related to the point chosen in the 3D model.    -   The user can evaluate model, zoom, pan, select a different        corner/vertex to verify/adjust alignment of the unverified        wireframe with that corner/vertex (e.g., bring up Projection        Views for that corner).    -   System can then move to the next roof corner in the sequence.    -   Corner being addressed is highlighted/marked in the overlay        view.

User reviews each Projection View,

-   -   if the unverified wireframe vertex being verified aligns exactly        on corresponding roof corner as pictured in the 2D image or as        shown on the point cloud comprising corresponding roof corner,        user indicates that no action is required;    -   if vertex does not align exactly with corresponding roof corner        in the 2D image and/or point cloud, user moves wireframe vertex        with mouse or via touch screen with finger or pen to align        wireframe vertex with corresponding roof corner.        -   The system can move the roof corner to an existing 3D point            or triangulate to create a new 3D point.        -   The vertex moves accordingly in each of the other visible            Projection Views.        -   The movement is logged (distance, angle, etc.).    -   System logs time spent on vertex.

User clicks “Next” to proceed to next corner/set of Projection Views.

Once all corners in sequence are reviewed (and adjusted, if necessary):

-   -   User indicates that sequence is done (such as by clicking        “complete” or similar in a screen dialogue, or just by        completing the last vertex in the sequence).        -   System can record that the unverified wireframe has been            adjusted along all relevant views to generate the verified            wireframe.        -   System can record total time user spent verifying and            adjusting each unverified wireframe to generate the verified            wireframe.    -   Alternative: User can click “Next” to restart sequence or select        “Back” or similar instructions to revisit the sequence.    -   Alternative: User can click on another vertex to review

If triangulation is determined to require substantial resources, analternative path can be:

-   -   if unverified wireframe vertex is not exactly on roof corner of        2D image or at the relevant part of a point cloud, the user can        move the vertex with, e.g., a mouse or via a finger/stylus, that        either:        -   ‘snaps to grid’ to a 3D point on the roof corner; or        -   free form moves to a visual roof corner (possibly indicating            via holding down shift key or clicking free-form button).            -   The system can triangulate the new 3D point on the                vertex, and projects the point on all Projection Views.

Add vertex workflow:

-   -   Reviews fit and completeness of wireframe on Overlay Model:        -   If corner is missing, user zooms into model and selects            point on model to add 3D point.            -   System displays Projection Views of point selected with                point back projected into the views, user adjusts point                to lay on corner across views.            -   Once point created, user can continue adding 3D points,                or exit the add mode.            -   Once complete, user can enter Connect mode and select                sequence of corners in model to connect with new lines.                -   Lines are drawn as corners are selected.            -   User can enter Delete mode to delete lines or points.

Inspection Mode

-   -   User can evaluate model, zoom, pan, select a point or corner to        view, and the Projection Views are displayed for that point.    -   In one mode, camera pose views are displayed above model and        user can select which views should go in Projection Views.        -   In one mode, when point is selected on a plane the plan is            highlights in the 3D model and the information (e.g., area,            diameter, time, etc.) area of the plane can be displayed in            the Status Pane.

Data and logs associated with the unverified wireframe generation,verification and/or adjustment session can be stored locally by the useror all or some aspects of the inventive process can be stored in thecloud, together or separate from wireframe and point cloud data.

The inventive system provides alternative ways to inspect or visualizeaspects of the unverified wireframe of the structure or element ofinterest. As described earlier, the system can provide a stacked layerview in which different information or aspects are provided as layers ontop of other layers or objects. In this view the user can see how thelayers relate (e.g., when referenced/scaled to the same coordinatesystem) such as to project the unverified wireframe on top of a pointcloud, project dimensions on top of a wireframe or point cloud or other3D structure, project a conditions layer on top of a wireframe or pointcloud or other 3D structure. Additionally, the system can providelayered lines or vertices or other structural components. The layeringapproach enables a user to select or deselect a specific layer or set oflayers. This capability can be extended to support groups of components,possibly connected to form polygons, and enable the user to move orinvestigate polygonal regions.

In further implementations, the verification methodology herein canprovide a view in which one aspect of the structure or element ofinterest is projected onto another view of the structure of interest.This can be represented by projected aspects of the unverified wireframeonto the 2D image(s) and/or point cloud or the 2D images onto thewireframe. For example, the vertices from the unverified wireframe canbe projected onto one or more 2D images. Similarly, one or more of the2D images could be projected onto either or both of the first orverified wireframe in order to provide a photo-realistic aspect to thepresented information. The system can enable movement of the projectedimage that provides a smooth transition and seemingly photo-realisticrepresentation of the information presented to the user.

The inventive system can provide a horizontally layered view in whichtwo structures of interest or portions of the same structure or elementof interest are presented side by side. Both layers can be seen at thesame time, which can enable comparisons between the layers. If there isa time aspect related to a structure, for example the date or time whenthe representation was created, the system can provide a time basedcomparison across the layers. For example, if verified wireframe Areflects the state of the structure at time 1 and the verified wireframeB represents the state of the structure of time 2, then the systemenables comparison of the structure across the timeframe. The system canbe configured to provide more than two horizontal layers, in furtherimplementations.

The inventive system can also provide a merged view in which it canrepresent two different integrated representations (e.g., wireframe,point cloud, image, model, etc.) and optionally highlight thesimilarities or differences on the layers. For example, the system couldmerge a point cloud representing the as-built structure with a modelrepresenting the as-designed structure, and highlight the differences orsimilarities in the two representations. Other examples include merginga layer generated point cloud with a photogrammetry generated pointcloud, or merging point clouds representing different parts of astructure or set of structures. Merging point clouds is also called“registering.”

In other aspects, the invention can provide a view optimized for quickvisualization of the overall structure of the wireframe or underlyingstructures as well as verification of accurate placement of wireframevertices, edges, and properties on the different representations of thestructure. This can be used to quickly validate the status of theunverified wireframe at different stages of the verification process,including but not limited to after the system generates a unverifiedwireframe, after a manual QA process is applied, or after a ground truthversion of the wireframe is created. This view can comprise in someembodiments the unverified wireframe optionally projected in 2D over the2D flattened view of a 3D structure (e.g., point cloud, mesh, BIM Model,etc.) or over a 2D image or over a 2D orthomosaic image. This view canbe optimized to ensure the structure is intact, properties (e.g., likeedge type, condition type, etc.) are assigned correctly and that thereare not extra vertices or missing vertices. The system can displayzoomed and possibly cropped images from one or more views of each vertexon the structure or element of interest. Such a view can enable the userto quickly view each vertex and know that it has been properly placedonto the exact position of that features/corner on the images. The viewcan take the form of a grid. The presentation to the user can beoptimized to present all or substantially all cropped images and set ofvertices in a consolidated view for quick visual processing. In suchcases the system can allow for application of a quality assuranceprocess for as little as 5 seconds or 10 seconds or 20 seconds or 30seconds or 1 minutes or 2 minutes.

In some cases, the one or more 2D images presented to the user can beplaced in a grid format. In cases in which the projected vertices arenot representative of as-built the user can press a control related tothe image to enter the inventive tool optionally opened up to thatvertex in adjust mode. The verification process can be configured topresent only 2D images or parts of the point cloud containing featuresor elements that were marked or otherwise noted by the user as requiringadjustment in the unverified wireframe. Such quick view system providescontrols to indicate that the wireframe has been verified and, thereforeis accurate and having user confidence.

In some aspects, the adjusting or points in the wireframe can beconsidered “rough adjustment” and could be in a 20 cm range or accuracy,or 10 cm, or 5 cm, or 2 cm. The process involves confirming theunverified wireframe is substantially complete at the start of theverification process, such as having no missing edges and no missingparts of the structure or element of interest. To accomplish this thesystem is configured to allow the user to zoom, rotate, manipulatevertices, edges and surface areas (using, e.g., Create, Update, Delete)in a 3D mode. Once the unverified wireframe is structurally sound eachvertex can be verified in the correct location by projecting a vertex onthe corresponding image or corresponding location on the point cloud.

The inventive systems can provide an improved approach to conductingaccuracy studies. Typically, accuracy studies involve one partymeasuring a scene using a first methodology, e.g., tape measure,documenting the results, and a second party measuring the same scenewith a second methodology, e.g., laser scanner or drone capture, etc.,and then comparing the results. Such accuracy studies can often belimited in effectiveness because it can be difficult to determinewhether the measurements were taken from the same endpoints in thestructure or element of interest. Without removing or reducing thatendpoint uncertainty, there can be lower user confidence in the accuracystudy results. One way to reduce this uncertainty is to mark the sceneand ask both parties to set their measurement endpoints on those marks.Sometimes the scenes cannot be reliability marked (difficult materials,customer not allowing marks, etc.). Additionally, there still is issueof confirming that the measurements were actually taken from thoseendpoints.

In regard to such accuracy studies, the inventive processes and systemsenable improvements in accuracy studies. For example, in the case ofaccuracy study comparing drones to hand measurements, the improvedapproach would be a “fly first and verify” approach. A structure orelement of interest having structural features of interest is selected.A drone would then be used to acquire a plurality of images of thestructure or element of interest. A wireframe would then be generatedand verified according to the methodology herein. A report would then begenerated with information that indicates in the associated imagesexactly where each endpoint was taken for each measurement or otherrelevant output derivable from the wireframe. This report or worksheetcould be produced before a worker (e.g., a surveyor) begins measuringthe structure or element of interest in person. The report or worksheetcan then be used as a template to capture measurement information inperson and to confirm that the in-person measurements conform to thecorresponding information generated from the images. This would enablethe person visiting the site to pick the exact same endpoints so as tobetter understand the accuracy of the image-derived measurementinformation. If the measurements that are provided on the report areaccurate, that fact could be noted and such information could beprovided for use. If the in-person derived measurements are not thesame, the person taking the in-person measurements could generatefurther information about the reasons why the measurements do not match.Information generated from such accuracy studies can be used to improvethe overall verification process, such as being incorporated intomachine learning or user training modules.

Referring to FIG. 1, shown is a flow diagram illustrating an example ofa wireframe verification process 100. Beginning at 103, 2D images of astructure or element of interest are provided that are derived from oneor more single passive image capture devices. The 2D images includeoverlapping 2D images with one or more structural aspects of thestructure or element of interest, and can alternatively be derived fromvideo. In addition, 3D representations of the structure or element ofinterest are provided at 106. The 3D representation can comprise or bederived from, e.g., point cloud information, surface mesh information,computer aided design (CAD) model information or building informationmodeling (BIM) model incorporating the structure or element of interest.

At 109, an unverified wireframe is generated from the 2D images and/orthe 3D representation. The unverified wireframe can be projected on 2Dimages, 3D representations, or a combination of both. The 2D images usedfor verification of the wireframe can include 2D images that were usedto generate the unverified wireframe. In some cases, the 2D images usedfor verification of the wireframe are separate from the 2D images usedto generate the wireframe (or a point cloud used to generate thewireframe). With the wireframe projected over the image or point cloudat 112, endpoints and edges can be compared to features of theunderlying 2D image or 3D representation to ensure confidence in thefinal verified wireframe. If adjustments are made at 115, then thechanges can be propagated through all of the 2D images used to verifythe unverified wireframe at 118. Confirmation of whether additional 2Dimages are available for verification can then be checked at 121. Ifthere is not adjustment, then the workflow passes to 121.

If there is another 2D image or 3D representation that is being used forverification, then the workflow returns to 112 where the unverifiedwireframe is projected on the next 2D image or 3D representation. Thecomparison of the unverified wireframe can, in some implementations,continue until verification with all of the 2D images and/or 3Drepresentation is complete at 121. Alternatively, the verification canbe completed after only some, such as one or one or more of the 2Dand/or 3D representation is complete at 121. At that time, the verifiedwireframe is produced at 124. The verified wireframe can be used with ahigh level of confidence. As noted, the verified wireframe is notnecessarily an accurate wireframe, but the user will nonetheless haveconfidence that the verified wireframe and any associated output is oris not accurate.

Referring now to FIG. 2, shown is an example of a machine 200 that maybe utilized for the wireframe verification methodology disclosed herein.The machine 200 can be a computing device 203 or other processingdevice, which includes at least one processor circuit, for example,having a processor 212 and a memory 215, both of which are coupled to alocal interface 218. To this end, the computing device(s) 203 maycomprise, for example, a server computer, mobile computing device (e.g.,laptop, tablet, smart phone, etc.) or any other system providingcomputing capability. The computing device(s) 203 may include, forexample, one or more display or touch screen devices and variousperipheral devices. Even though the computing device 203 is referred toin the singular, it is understood that a plurality of computing devices203 may be employed in the various arrangements as described above. Thelocal interface 218 may comprise, for example, a data bus with anaccompanying address/control bus or other bus structure as can beappreciated.

Stored in the memory 215 are both data and several components that areexecutable by the processor 212. In particular, stored in the memory 215and executable by the processor 212 include a wireframe verificationapplication 221 and potentially other applications. Also stored in thememory 215 may be a data store 224 and other data. The data stored inthe data store 224, for example, is associated with the operation of thevarious applications and/or functional entities described below. Forexample, the data store may include databases, object libraries, andother data or information as can be understood. In addition, anoperating system 227 may be stored in the memory 215 and executable bythe processor 212. The data store 224 may be may be located in a singlecomputing device or may be dispersed among many different devices. Thecomponents executed on the computing device 203 include, for example,the wireframe verification application 221 and other systems,applications, services, processes, engines, or functionality notdiscussed in detail herein. It is understood that there may be otherapplications that are stored in the memory 215 and are executable by theprocessor 212 as can be appreciated. Where any component discussedherein is implemented in the form of software, any one of a number ofprogramming languages may be employed.

A number of software components are stored in the memory 215 and areexecutable by the processor 212. In this respect, the term “executable”means a program file that is in a form that can ultimately be run by theprocessor 212. Examples of executable programs may be, for example, acompiled program that can be translated into machine instructions in aformat that can be loaded into a random access portion of the memory 215and run by the processor 212, source code that may be expressed inproper format such as object code that is capable of being loaded into arandom access portion of the memory 215 and executed by the processor212, or source code that may be interpreted by another executableprogram to generate instructions in a random access portion of thememory 215 to be executed by the processor 212, etc. An executableprogram may be stored in any portion or component of the memory 215including, for example, random access memory (RAM), read-only memory(ROM), hard drive, solid-state drive, USB flash drive, memory card,optical disc such as compact disc (CD) or digital versatile disc (DVD),floppy disk, magnetic tape, or other memory components.

Also, the processor 212 may represent multiple processors 212 and thememory 215 may represent multiple memories 215 that operate in parallelprocessing circuits, respectively. In such a case, the local interface218 may be an appropriate network that facilitates communication betweenany two of the multiple processors 212, between any processor 212 andany of the memories 215, or between any two of the memories 215, etc.The local interface 218 may comprise additional systems designed tocoordinate this communication, including, for example, performing loadbalancing. The processor 212 may be of electrical or of some otheravailable construction.

Although the wireframe verification application 221, and other varioussystems described herein, may be embodied in software or instructionsexecuted by general purpose hardware as discussed above, as analternative the same may also be embodied in dedicated hardware or acombination of software/general purpose hardware and dedicated hardware.If embodied in dedicated hardware, each can be implemented as a circuitor state machine that employs any one of or a combination of a number oftechnologies. These technologies may include, but are not limited to,discrete logic circuits having logic gates for implementing variouslogic functions upon an application of one or more data signals,application specific integrated circuits having appropriate logic gates,or other components, etc. Such technologies are generally well known bythose skilled in the art and, consequently, are not described in detailherein.

Any logic or application described herein, including the wireframeverification application 221, that comprises software or instructionscan be embodied in any non-transitory computer-readable medium for useby or in connection with an instruction execution system such as, forexample, a processor 212 in a computer system or other system. In thissense, the logic may comprise, for example, statements includinginstructions and declarations that can be fetched from thecomputer-readable medium and executed by the instruction executionsystem. The flow diagram of FIG. 1 shows an example of the architecture,functionality, and operation of possible implementations of a wireframeverification application 221. In this regard, each block can represent amodule, segment, or portion of code, which comprises one or moreexecutable instructions for implementing the specified logicalfunction(s). It should also be noted that in some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in FIG. 1. For example, two blocks shown in succession inFIG. 1 may in fact be executed substantially concurrently or the blocksmay sometimes be executed in a different or reverse order, dependingupon the functionality involved. Alternate implementations are includedwithin the scope of the preferred embodiment of the present disclosurein which functions may be executed out of order from that shown ordiscussed, including substantially concurrently or in reverse order,depending on the functionality involved, as would be understood by thosereasonably skilled in the art of the present disclosure.

Communication media appropriate for use in or with the inventions of thepresent disclosure may be exemplified by computer-readable instructions,data structures, program modules, or other data stored on non-transientcomputer-readable media, and may include any information-delivery media.The instructions and data structures stored on the non-transientcomputer-readable media may be transmitted as a modulated data signal tothe computer or server on which the computer-implemented methods of thepresent disclosure are executed. A “modulated data signal” may be asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media may include wired media such asa wired network or direct-wired connection, and wireless media such asacoustic, radio frequency (RF), microwave, infrared (IR) and otherwireless media. The term “computer-readable media” as used herein mayinclude both local non-transient storage media and remote non-transientstorage media connected to the information processors usingcommunication media such as the internet. Non-transientcomputer-readable media do not include mere signals or modulated carrierwaves, but include the storage media that form the source for suchsignals.

In the context of the present disclosure, a “computer-readable medium”can be any medium that can contain, store, or maintain the logic orapplication described herein for use by or in connection with theinstruction execution system. The computer-readable medium can compriseany one of many physical media such as, for example, electronic,magnetic, optical, electromagnetic, infrared, or semiconductor media.More specific examples of a suitable computer-readable medium wouldinclude, but are not limited to, magnetic tapes, magnetic floppydiskettes, magnetic hard drives, memory cards, solid-state drives, USBflash drives, or optical discs. Also, the computer-readable medium maybe a random access memory (RAM) including, for example, static randomaccess memory (SRAM) and dynamic random access memory (DRAM), ormagnetic random access memory (MRAM). In addition, the computer-readablemedium may be a read-only memory (ROM), a programmable read-only memory(PROM), an erasable programmable read-only memory (EPROM), anelectrically erasable programmable read-only memory (EEPROM), or othertype of memory device.

At this time, there is little distinction left between hardware andsoftware implementations of aspects of systems; the use of hardware orsoftware is generally (but not always, in that in certain contexts thechoice between hardware and software can become significant) a designchoice representing cost vs. efficiency tradeoffs. There are variousinformation-processing vehicles by which processes and/or systems and/orother technologies described herein may be implemented, e.g., hardware,software, and/or firmware, and that the preferred vehicle may vary withthe context in which the processes and/or systems and/or othertechnologies are deployed. For example, if an implementer determinesthat speed and accuracy are paramount, the implementer may opt for amainly hardware and/or firmware vehicle; if flexibility is paramount,the implementer may opt for a mainly software implementation; or, yetagain alternatively, the implementer may opt for some combination ofhardware, software, and/or firmware.

The foregoing detailed description has set forth various aspects of thedevices and/or processes for system configuration via the use of blockdiagrams, flowcharts, and/or examples. Insofar as such block diagrams,flowcharts, and/or examples contain one or more functions and/oroperations, it will be understood by those within the art that eachfunction and/or operation within such block diagrams, flowcharts, orexamples can be implemented, individually and/or collectively, by a widerange of hardware, software, firmware, or virtually any combinationthereof. In one embodiment, several portions of the subject matterdescribed herein may be implemented via Application Specific IntegratedCircuits (ASICs), Field Programmable Gate Arrays (FPGAs), digital signalprocessors (DSPs), or other integrated formats. However, those skilledin the art will recognize that some aspects of the aspects disclosedherein, in whole or in part, can be equivalently implemented inintegrated circuits, as one or more computer programs running on one ormore computers, e.g., as one or more programs running on one or morecomputer systems, as one or more programs running on one or moreprocessors, e.g., as one or more programs running on one or moremicroprocessors, as firmware, or as virtually any combination thereof,and that designing the circuitry and/or writing the code for thesoftware and or firmware would be well within the skill of one of skillin the art in light of this disclosure. In addition, those skilled inthe art will appreciate that the mechanisms of the subject matterdescribed herein are capable of being distributed as a program productin a variety of forms, and that an illustrative embodiment of thesubject matter described herein applies regardless of the particulartype of signal bearing medium used to actually carry out thedistribution. Examples of a signal-bearing medium include, but are notlimited to, the following: a recordable type medium such as a floppydisk, a hard disk drive, a CD, a DVD, a digital tape, a computer memory,etc.; and a remote non-transitory storage medium accessed using atransmission type medium such as a digital and/or an analogcommunication medium (e.g., a fiber optic cable, a waveguide, a wiredcommunications link, a wireless communication link, etc.), for example aserver accessed via the internet.

Those skilled in the art will recognize that it is common within the artto describe devices and/or processes in the fashion set forth herein,and thereafter use engineering practices to integrate such describeddevices and/or processes into data-processing systems. That is, at leasta portion of the devices and/or processes described herein can beintegrated into a data processing system via a reasonable amount ofexperimentation. Those having skill in the art will recognize that atypical data processing system generally includes one or more of asystem unit housing, a video display device, a memory such as volatileand non-volatile memory, processors such as microprocessors and digitalsignal processors, computational entities such as operating systems,drivers, graphical user interfaces, and applications programs, one ormore interaction devices, such as a touch pad or screen, and/or controlsystems including feedback loops and control motors, e.g., feedback forsensing position and/or velocity; control motors for moving and/oradjusting components and/or quantities. A typical data processing systemmay be implemented utilizing any suitable commercially availablecomponents, such as those typically found in datacomputing/communication and/or network computing/communication systems.

The herein-described subject matter sometimes illustrates differentcomponents contained within, or connected with, different othercomponents. It is to be understood that such depicted architectures aremerely examples, and that in fact many other architectures can beimplemented which achieve the same functionality. In a conceptual sense,any arrangement of components to achieve the same functionality iseffectively “associated” such that the desired functionality isachieved. Hence, any two components herein combined to achieve aparticular functionality can be seen as “associated with” each othersuch that the desired functionality is achieved, irrespective ofarchitectures or intermedial components. Likewise, any two components soassociated can also be viewed as being “operably connected”, or“operably coupled”, to each other to achieve the desired functionality,and any two components capable of being so associated can also be viewedas being “operably couplable”, to each other to achieve the desiredfunctionality. Specific examples of operably couplable include but arenot limited to physically mateable and/or physically interactingcomponents and/or wirelessly interactable and/or wirelessly interactingcomponents and/or logically interacting and/or logically interactablecomponents.

As described above, the exemplary aspects have been described andillustrated in the drawings and the specification. The exemplary aspectswere chosen and described in order to explain certain principles of theinvention and their practical application, to thereby enable othersskilled in the art to make and utilize various exemplary aspects of thepresent invention, as well as various alternatives and modificationsthereof. As is evident from the foregoing description, certain aspectsof the present invention are not limited by the particular details ofthe examples illustrated herein, and it is therefore contemplated thatother modifications and applications, or equivalents thereof, will occurto those skilled in the art. Many changes, modifications, variations andother uses and applications of the present construction will, however,become apparent to those skilled in the art after considering thespecification and the accompanying drawings. All such changes,modifications, variations and other uses and applications which do notdepart from the spirit and scope of the invention are deemed to becovered by the invention.

What is claimed is:
 1. A method of generating a verified wireframe of astructure or element of interest comprising: a. generating, by acomputer or a user, an unverified wireframe corresponding to at leastpart of a structure or element of interest comprising one or morestructural aspects of interest, wherein: the unverified wireframe:
 1. isderived from a plurality of overlapping 2D images of the structure orelement of interest, wherein the 2D images are generated from a passiveimage capture device, and incorporates one or more of the structuralaspects; and
 2. comprises one or more features that correspond to theone or more structural aspects; and b. projecting the unverifiedwireframe over either or both of: i. one or more 2D images selected fromthe plurality of overlapping 2D images, wherein each of the selected 2Dimages incorporates at least some of the one or more structural aspects;or ii. a point cloud derived from the plurality of overlapping 2Dimages, wherein the point cloud incorporates at least some of the one ormore structural aspects; and c. adjusting, by either or both of thecomputer or user, at least part of the unverified wireframe, wherein theadjusting is by alignment of at least one of the one or more unverifiedwireframe features with a corresponding structural aspect visible ineach of the one or more selected 2D images or the point cloud over whichthe unverified wireframe is projected, or indicating, by either or bothof the computer or user, acceptance of the unverified wireframe withoutadjustment, thereby generating a verified wireframe comprising all orpart of the structure or element of interest.
 2. The method of claim 1,wherein the structure or element of interest comprises all or part of aroof, and the one or more structural aspects of interest comprises aroof area, a roof corner, a roof pitch, a roof edge, a roof gutter, aroof gable, a dormer, or a skylight.
 3. The method of claim 1, whereinat least part of the adjusting is conducted by a user and a plurality of2D images are selected for presentation to the user, wherein each of theplurality of 2D images are selected by the user or the computer topresent a different view of the one or more structural aspects ofinterest to the user.
 4. The method of claim 3, wherein, when the useraligns at least one unverified wireframe feature with a correspondingstructural aspect of interest on a presented 2D image, modifications tothe wireframe resulting from the alignment are visible to the user on atleast one other 2D image in the plurality of presented 2D images.
 5. Themethod of claim 1, wherein measurement information can be derived fromthe verified wireframe, and wherein the measurement information is lessthan 5% different from an actual measurement value for the correspondingstructural aspect.
 6. The method of claim 1, wherein measurementinformation can be derived from the verified wireframe, and wherein themeasurement information is less than 1% different from an actualmeasurement value for the corresponding structural aspect.
 7. The methodof claim 1, wherein: a. the adjusting step is associated with one ormore rules associated with the one or more structural aspects ofinterest; and b. each of the one or more rules defines an allowedpositioning for the one or more structural aspects, a prohibitedpositioning for the one or more structural aspects, or both an allowedand a prohibited positioning for the one or more structural aspects. 8.The method of claim 7, wherein: a. when a user generated adjustmentconforms to an allowed positioning for the structural aspect theadjustment will be maintained by the computer; or b. when the usergenerated adjustment conforms to a prohibited positioning for thestructural aspect the adjustment will be rejected by the computer. 9.The method of claim 1, wherein the wireframe verification process isgenerated by a user, and during the projection and adjustment stepseither or both of visual or audible instructions are provided to theuser by the computer.
 10. The method of claim 1, wherein informationabout one or more of the projection or the adjustment of the unverifiedwireframe is collected by the computer for incorporation into a machinelearning process.
 11. A system, comprising: at least one computingdevice comprising a processor; and a wireframe verification applicationstored in memory, where execution of the wireframe verificationapplication by the processor causes the at least one computing deviceto: a. generate an unverified wireframe corresponding to at least partof a structure or element of interest comprising one or more structuralaspects of interest, wherein the unverified wireframe: i. is derivedfrom a plurality of overlapping 2D images of the structure or element ofinterest, wherein the 2D images are generated from a passive imagecapture device, and incorporates one or more of the structural aspects;and ii. comprises one or more features that correspond to the one ormore structural aspects; and b. project the unverified wireframe overeither or both of: i. one or more 2D images selected from the pluralityof overlapping 2D images, wherein each of the selected 2D imagesincorporates at least some of the one or more structural aspects; or ii.a point cloud derived from the plurality of overlapping 2D images,wherein the point cloud incorporates at least some of the one or morestructural aspects; and c. adjust at least part of the unverifiedwireframe, wherein the adjusting is by alignment of at least one of theone or more unverified wireframe features with a correspondingstructural aspect visible in each of the one or more selected 2D imagesor the point cloud over which the unverified wireframe is projected, orindicate acceptance of the unverified wireframe without adjustment,thereby generating a verified wireframe comprising all or part of thestructure or element of interest.
 12. The system of claim 11, whereinthe unverified wireframe is generated based upon user input.
 13. Thesystem of claim 11, wherein at least part of the adjusting is conductedbased upon a user input and a plurality of 2D images are selected forpresentation to the user, wherein each of the plurality of 2D images areselected by the user or the computer to present a different view of theone or more structural aspects of interest to the user.
 14. The systemof claim 13, wherein, when the user aligns at least one unverifiedwireframe feature with a corresponding structural aspect of interest ona presented 2D image, modifications to the wireframe resulting from thealignment are visible to the user on at least one other 2D image in theplurality of presented 2D images.
 15. The system of claim 11, whereinmeasurement information can be derived from the verified wireframe, andwherein the measurement information is less than 5% different from anactual measurement value for the corresponding structural aspect. 16.The system of claim 11, wherein: a. the adjusting step is associatedwith one or more rules associated with the one or more structuralaspects of interest; and b. each of the one or more rules defines anallowed positioning for the one or more structural aspects, a prohibitedpositioning for the one or more structural aspects, or both an allowedand a prohibited positioning for the one or more structural aspects. 17.The system of claim 16, wherein: a. when a user generated adjustmentconforms to an allowed positioning for the structural aspect theadjustment will be maintained by the wireframe verification application;or b. when the user generated adjustment conforms to a prohibitedpositioning for the structural aspect the adjustment will be rejected bythe wireframe verification application.
 18. The system of claim 11,wherein the wireframe verification application presents either or bothof visual or audible instructions to the user during the projection andadjustment steps.
 19. The system of claim 11, wherein information aboutone or more of the projection or the adjustment of the unverifiedwireframe is collected by the wireframe verification application andstored in the memory for incorporation into a machine learning process.20. The system of claim 11, wherein the indication of acceptance isbased upon a user input.